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Abstract:  Optically  multiplexed  imaging  is  the  process  by  which  multiple 
images  are  overlaid  on  a  single  image  surface.  Uniquely  encoding  the 
discrete  images  allows  scene  reconstruction  from  multiplexed  images  via 
post  processing.  We  describe  a  class  of  optical  systems  that  can  achieve 
high  density  image  multiplexing  through  a  novel  division  of  aperture 
technique.  Fundamental  design  considerations  and  performance  attributes 
for  this  sensor  architecture  are  discussed.  A  number  of  spatial  and  temporal 
encoding  methods  are  presented  including  point  spread  function 
engineering,  amplitude  modulation,  and  image  shifting.  Results  from  a 
prototype  five -channel  sensor  are  presented  using  three  different  encoding 
methods  in  sparse-scene  star  tracking  demonstration.  A  six-channel 
optically  multiplexed  prototype  sensor  is  used  to  reconstruct  imagery  from 
information  rich  dense  scenes  through  dynamic  image  shifting. 
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(080.3620)  Lens  system  design;  (220.2740)  Geometric  optical  design;  (220.4830)  Systems 
design. 


References  and  links 

1 .  M.  D.  Stenner,  P.  Shankar,  and  M.  A.  Neifeld,  “Wide-Field  Feature-Specific  Imaging,”  in  Frontiers  in  Optics 
2007,  OSA  Technical  Digest  (Optical  Society  of  America,  2007),  paper  FMJ2. 

2.  R.  F.  Marcia,  C.  Kim,  C.  Eldeniz,  J.  Kim,  D.  J.  Brady,  and  R.  M.  Willett,  “Superimposed  video  disambiguation 
for  increased  field  of  view,”  Opt.  Express  16(21),  16352-16363  (2008). 

3.  S.  Uttam,  N.  A.  Goodman,  M.  A.  Neifeld,  C.  Kim,  R.  John,  J.  Kim,  and  D.  Brady,  “Optically  multiplexed 
imaging  with  superposition  space  tracking,”  Opt.  Express  17(3),  1691-1713  (2009). 

4.  V.  Treeapom,  A.  Ashok,  and  M.  A.  Neifeld,  “Increased  field  of  view  through  optical  multiplexing,”  Opt. 

Express  18(21),  22432-22445  (2010). 

5.  R.  Horisaki  and  J.  Tanida,  “Multi-channel  data  acquisition  using  multiplexed  imaging  with  spatial  encoding,” 
Opt.  Express  18(22),  23041-23053  (2010). 

6.  C.  Y.  Chen,  T.  T.  Yang,  and  W.  S.  Sun,  “Optics  system  design  applying  a  micro-prism  array  of  a  single  lens 
stereo  image  pair,”  Opt.  Express  16(20),  15495-15505  (2008). 

7.  A.  Mahalanobis,  M.  Neifeld,  V.  K.  Bhagavatula,  T.  Haberfelde,  and  D.  Brady,  “Off-axis  sparse  aperture  imaging 
using  phase  optimization  techniques  for  application  in  wide-area  imaging  systems,”  Appl.  Opt.  48(28),  5212- 
5224  (2009). 

8.  A.  Daniels,  “Infrared  systems  -  technology  &  design,”  SPIE  Short  Course  SC835,  279  (2015). 

9.  J.  Sasian,  “Interpretation  of  pupil  aberrations  in  imaging  systems,”  Proc.  SPIE  6342,  634208  (2006). 

10.  D.  Shafer,  “Aberration  Theory  and  the  Meaning  of  Life,”  Proc.  SPIE  554,  25  (1986). 

11.  R.  Gupta,  P.  Indyk,  E.  Price,  and  Y.  Rachlin,  “Compressive  sensing  with  local  geometric  features,”  Proc.  of  the 
27th  annual  ACM  symposium  on  computational  geometry,  87-98,  ACM  (201 1). 

12.  Y.  Rachlin,  V.  Shah,  R.  H.  Shepard,  and  T.  Shih,  “Dynamic  optically  multiplexed  imaging,”  Proc.  SPIE  9600, 
96003  (2015). 

13.  S.  G.  Mallat  and  Z.  Zhang,  “Matching  pursuits  with  time-frequency  dictionaries,”  Signal  Processing,  IEEE 
Transactions  on  41(12),  3397-3415  (1993). 

14.  E.  J.  Candes  and  T.  Tao,  “Near-optimal  signal  recovery  from  random  projections:  universal  encoding 
strategies?”  IEEE  Trans.  Inf  Theory  52(12),  5406-5425  (2006). 

15.  D.  L.  Donoho,  “Compressed  sensing,”  IEEE  Trans.  Inf.  Theory  52(4),  1289-1306  (2006). 

16.  D.  L.  Donoho,  A.  Maleki,  and  A.  Montanari,  “Message-passing  algorithms  for  compressed  sensing,”  Proc.  Natl. 
Acad.  Sci.  U.S.A.  106(45),  18914-18919  (2009). 


#250668 
©2015  OSA 


Received  22  Sep  2015;  revised  6  Nov  2015;  accepted  7  Nov  2015;  published  23  Nov  2015 
30  Nov  2015  I  Vol.  23,  No.  24  |  DOI:10.1364/OE.23.031419  |  OPTICS  EXPRESS  31419 


17.  Y.  C.  Pati,  R.  Rezaiifar,  and  P.  S.  Krishnaprasad,  “Orthogonal  matching  pursuit:  Recursive  function 

approximation  with  applications  to  wavelet  decomposition,”  Proc.  27th  Asilomar  Conference  on  Signals, 

Systems  and  Computers.  IEEE  Computer  Society  Press,  40  (1993). 

18.  C.  C.  Paige  and  M.  A.  Saunders,  “LSQR:  An  algorithm  for  sparse  linear  equations  and  sparse  least  squares,” 

TOMS  8(1),  43-71  (1982). 

1.  Introduction 

In  many  imaging  systems,  the  deteetor  under  samples  the  optieal  point  spread  funetion 
produeing  an  image  resolution  that  is  limited  by  the  number  of  pixels  on  the  foeal  plane  array 
(FPA).  This  introduees  a  fundamental  tradeoff  between  the  field  of  view  (FOV)  of  a  sensor 
and  its  sampling  resolution  in  objeet  spaee.  Methods  have  been  developed  to  ereate  wide 
FOV  systems  with  enhaneed  pixel  resolution  by  stitehing  together  images  from  multiple 
sensors,  by  seanning  a  single  sensor  aeross  the  seene,  and  through  super-resolution  teehniques 
that  eombine  a  series  of  images  with  sub-pixel  shifts.  Like  the  vast  majority  of  sensors  used 
today,  these  systems  are  based  on  the  prineiple  that  at  any  instant  in  time  eaeh  pixel  views 
only  a  single  point  in  objeet  spaee.  Therefore,  inereasing  the  spatial  resolution  requires  either 
more  pixels  or  more  time. 

Optieally  multiplexed  imaging  is  based  on  the  prineiple  that  a  single  pixel  ean  observe 
multiple  objeet  points  simultaneously.  This  has  been  investigated  in  design  arehiteetures  that 
use  multiple  lenses  to  form  images  on  a  single  FPA  [1],  a  easeade  of  beam  splitting  elements 
to  divert  multiple  fields  of  view  into  a  single  lens  [2-5],  and  by  plaeing  an  interleaved  array 
of  sub-aperture  miero-prisms  in  front  of  a  single  lens  [6,7].  In  this  paper  we  present  results 
using  a  new  optieal  design  arehiteeture  based  on  a  division  of  aperture  teehnique  to  divide  the 
pupil  area  of  a  single  lens  into  a  number  of  independent  imaging  ehannels.  This  method,  first 
introdueed  in  [12],  offers  advantages  over  prior  approaehes  through  its  flexibility  to 
individually  direet  and  eneode  the  optieal  ehannels,  and  it  yields  a  signifieant  volume 
advantage  in  systems  with  a  high  degree  of  multiplexing. 

The  image  formed  by  an  optieally  multiplexed  system  is  the  superposition  of  multiple 
images  formed  by  diserete  imaging  ehannels.  A  seene  reeonstmetion  proeess  is  required  to 
separate  the  individual  ehannel  images  and  reeonstruet  an  estimate  of  the  seene.  One  broad 
eategory  of  disambiguation  strategies  is  to  use  temporal  eneoding  to  eonstmet  a  fully 
determined  system  of  equations  from  a  number  of  uniquely  eneoded  samples  of  the  seene. 
This  strategy  is  useful  for  dense  information-rieh  seenes  and  has  been  previously  aehieved  by 
using  shutters  to  attenuate  individual  imaging  ehannels  [4]  or  by  using  a  single  moving 
element  to  shift  one  of  two  ehannels  between  samples  [2].  Temporal  eneoding  methods 
explored  through  our  researeh  inelude  aetive  shutter-based  method,  an  aetive  method  of  rapid 
and  preeise  imaging  shifting  [12],  and  a  passive  method  that  uses  image  rotation  to  observe 
differential  image  shifts  in  a  moving  seene.  Another  broad  eategory  of  disambiguation 
strategies  has  applieations  in  sparse  seenes.  These  spatial  eneoding  methods  use  point  spread 
funetion  (PSF)  engineering  to  uniquely  eneode  eaeh  ehannel  image.  Spatial  image  eneoding 
methods  provide  the  advantage  of  disambiguation  of  a  multi-layered  image  from  a  single 
measurement  of  the  seene,  whieh  addresses  sparse  seene  applieations  sueh  as  point  souree 
deteetion  or  star  field  mapping. 

In  the  following  seetions  of  this  paper,  we  examine  the  elass  of  optieally  multiplexed 
imaging  systems  that  eombine  multiple  imaging  ehannels  into  the  entranee  pupil  of  a  single 
imaging  sub-system.  In  Seetion  2,  we  deseribe  the  two  fundamental  methods  of  pupil  division 
multiplexing,  division  of  amplitude  and  division  of  aperture,  and  we  diseuss  the  relative 
performanee  implieations  of  eaeh  method.  Seetion  3  establishes  the  fundamental  optieal 
design  eonsiderations  for  the  division  of  aperture  method.  Topies  inelude  eneoding  uniformity 
and  its  relationship  to  pupil  sampling,  refieetive  and  refraetive  multiplexing  assemblies.  In 
Seetion  4  methods  of  image  eneoding  and  deeoding  are  deseribed.  In  Seetion  5,  we  deseribe 
our  prototype  sensors:  a  pair  of  5-ehannel  designs  that  employ  three  different  eneoding 
meehanisms  (spatial  image  eneoding,  passive  temporal  eneoding  via  image  rotation  and  seene 
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motion,  and  active  temporal  encoding  via  channel  attenuation),  and  also  a  six-channel 
prototype  that  uses  piezeoelectric  actuators  for  rapid  and  precise  image  shifting.  Section  6 
presents  experimental  results  for  the  prototypes  in  sparse-scene  and  dense-scene  applications. 

2.  System  architecture  and  performance  attributes 

2.1  Multiplexing  architectures 

Consider  an  optically  multiplexed  imaging  system  composed  of  a  single  image  forming 
element,  which  we  will  refer  to  as  the  parent  lens,  and  a  multiplexing  element  that  directs 
multiple  discrete  imaging  channels  into  the  aperture  of  this  lens,  which  we  will  refer  to  as  the 
multiplexing  assembly.  Through  the  design  of  the  multiplexing  assembly,  these  channels  can 
be  directed  to  observe  overlapping,  adjacent,  or  separated  fields  of  view.  Every  optical  system 
has  a  finite  limit  on  the  amount  of  power  it  can  collect.  This  power  (O)  may  be  expressed  as 
the  product  of  the  transmission  between  the  object  and  image  planes  (r),  the  average  radiance 
of  the  scene  (T),  the  area  of  the  collection  aperture  (A),  and  the  solid  angle  of  the  field  of  view 
(Q). 

O  =  tLAQ  (1) 

The  product  AQ  is  known  as  the  throughput  or  etendue  of  system,  and  is  a  conserved 
quantity  that  may  be  expressed  as  various  combinations  of  the  system’s  focal  length,  aperture 
size,  F/#,  field  of  view,  or  detector  area.  The  basic  principle  of  an  optically  multiplexed 
imaging  system  is  to  increase  the  field  of  view  or  pixel  count  of  an  image  by  overlapping 
multiple  images  on  the  detector  plane  wherein  the  total  power  is  the  sum  of  power  from  each 
multiplexed  channel.  Under  the  condition  of  uniform  scene  radiance,  the  maximum  power 
collected  by  the  optical  system  becomes  a  conserved  quantity,  which  implies  that 
multiplexing  may  only  be  achieved  through  a  tradeoff  between  the  terms  in  Eq.  (1).  Thus, 
there  are  fundamentally  two  ways  to  achieve  image  multiplexing:  division  of  amplitude  and 
division  of  aperture,  which  are  illustrated  in  Fig.  1 . 

We  define  division  of  amplitude  as  the  multiplexing  technique  that  increases  the  solid 
angle  of  the  FOV  by  directing  different  portions  of  the  pupil’s  transmission  function  to  view 
different  regions  of  object  space.  By  Eq.  (2)  we  see  that  the  number  of  additional  FOV 
channels  {N)  used  to  increase  the  solid  angle  of  the  system  is  directly  related  to  the 
transmission  loss  of  each  channel,  assuming  uniform  division.  Dividing  the  pupil 
transmission  may  be  achieved  by  splitting  the  polychromatic  signal  intensity  between 
channels,  splitting  it  spectrally  so  that  each  channel  observes  a  different  wavelength  band,  or 
by  splitting  it  polarimetrically  such  that  a  different  polarization  state  is  observed  in  each 
channel. 


(2) 

A  multiplexed  imager  based  on  division  of  amplitude  might  be  designed  using  a  series  of 
cascaded  beam  splitters  and  mirrors  or  with  birefringent  prisms.  This  method  holds  the 
advantage  that  properly  sized  and  coated  elements  can  produce  multiplexing  with  a  high 
degree  of  irradiance  uniformity  between  the  channels  and  no  vignetting.  Additionally,  each 
channel  may  retain  its  full  aperture  area  and  will  therefore  have  the  same  F/#  and  optical 
resolution  as  the  parent  lens.  Flowever,  a  drawback  of  this  architecture  is  that  the  multiplexing 
elements  become  very  large.  Not  only  must  each  element  accommodate  the  full  pupil  of  the 
parent  lens,  but  the  elements  must  also  grow  progressively  larger  as  they  move  further  away 
from  the  entrance  pupil  to  avoid  vignetting  as  ray  bundles  diverge  across  the  FOV. 


#250668 
©2015  OSA 


Received  22  Sep  2015;  revised  6  Nov  2015;  accepted  7  Nov  2015;  published  23  Nov  2015 
30  Nov  2015  I  Vol.  23,  No.  24  |  001:10. 1364/OE.23.031419  |  OPTICS  EXPRESS  31421 


Fig.  1.  Pupil  division  strategies,  (a)  division  of  amplitude  using  a  cascade  of  beam  splitters,  (b) 
division  of  aperture  using  an  array  of  prisms. 

The  multiplexing  arehiteeture  we  define  as  division  of  aperture  is  the  primary  foeus  of  this 
paper.  In  this  teehnique  the  field  of  view  solid  angle  is  extended  by  a  faetor  of  N  by  dividing 
the  parent  lens’  pupil  area  and  redireeting  eaeh  ehannel  to  different  regions  of  objeet  spaee  as 
deseribed  in  Eq.  (3). 

(3) 

Division  of  aperture  holds  a  distinet  volume  advantage  over  division  of  pupil  transmission 
because  the  entire  multiplexing  assembly  may  be  as  small  as  the  pupil  of  the  parent  lens,  and 
any  additional  splitting,  folding,  or  encoding  elements  need  only  accommodate  a  sub-pupil 
region.  A  potential  compromise  is  that  the  smaller  aperture  area  (Af)  increases  the  channel  F- 
number  (F/#n)  causing  a  loss  of  optical  resolution.  Some  degree  of  resolution  loss  may  be 
mitigated  with  an  interleaved  multiplexing  element  that  directs  a  spatially  distributed 
sampling  of  the  pupil  to  each  channel  [7].  Flowever,  this  implementation  presents  the 
challenge  of  properly  phasing  the  elements  within  each  channel,  and  if  secondary  folding  or 
encoding  elements  are  required  much  of  the  volume  advantage  would  be  lost. 

2.2  Multiplexed  system  performance  attributes 

An  early  decision  in  the  design  process  is  the  selection  of  the  number  of  channels  to  be 
multiplexed.  Increasing  the  number  of  channels  might  increase  the  field  of  view  and  the 
number  of  pixels  recovered  in  the  final  image,  but  there  are  also  consequences  related  to 
image  quality  and  design  complexity.  Flere  we  describe  a  few  of  the  basic  effects  pertaining 
to  division  of  aperture  architectures. 

2.2.1  Aperture  and  field  of  view 

Beginning  with  first-order  optical  design  parameters  we  start  with  the  assumption  that  each 
multiplexed  channel  shares  the  same  parent  lens.  Thus,  the  effective  focal  length  of  each 
channel  (JIn)  will  be  equivalent  to  the  parent  lens  focal  lens  iflo)\ 

flN=flo-  (4) 

In  a  division  of  aperture  system  the  aperture  area  of  the  parent  lens  (Ao)  is  divided  equally 
among  N  channels 


(5) 
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which  can  be  used  to  approximate  the  effective  channel  aperture  diameter  (Jat)  and  effective 
channel  F-number  (F’#n)  as 


and 


(6) 


(7) 

The  fields  of  view  of  the  individual  imaging  channels  may  be  arranged  in  any 
configuration  in  object  space  to  view  either  continuous  or  discontinuous  regions  of  the  scene. 
Each  channel  will  have  a  FOV  equal  to  the  FOV  of  the  parent  lens 


^.N.yN  =  “  2Tan“’ 


2A 


(8) 


where  Oxo.yo  are  the  full  horizontal  and  vertical  FOV  of  the  parent  lens,  the  width  of  each  pixel 
in  the  focal  plane  is  {dp)  and  the  number  horizontal  or  vertical  pixels  in  the  camera  is  rix  and 
riy,  respectively.  The  solid  angle  of  the  total  field  of  view  is  directly  related  to  the  number  of 
multiplexed  channels. 

2.2.2  Resolution 

Whether  or  not  multiplexing  increases  the  resolution  of  the  disambiguated  image  depends  on 
whether  the  resolution  is  limited  by  the  detector  sampling  or  by  the  optical  point  spread 
function.  In  the  under-sampled  detector-limited  regime  the  resolution  of  the  multiplexed 
system  is  dictated  by  the  number  of  pixels  sampling  the  object  space,  Mpix,  which  increases 
directly  with  the  number  of  multiplexed  channels  as  shown  in  Eq.  (9). 

Mp,x,N=Nn^n^.-  (9) 

With  optically- limited  sampling  the  resolution  can  be  described  by  the  sensor’s  space- 
bandwidth  product,  Mopt,  which  is  defined  by  the  solid  angle  of  the  field  of  view  divided  by 
the  solid  angle  of  the  diffraction-limited  Airy  Disk 

Nd  e 

(10) 

A  1 


Thus,  the  resolution  of  the  multiplexed  system  ranges  from  the  resolution  of  the  parent  system 
(when  optically  limited)  up  to  N-times  the  pixel-count  of  the  camera  when  the  sensor  is 
detector  limited.  In  the  optically  limited  case  multiplexing  does  not  increase  resolution; 
however,  depending  on  the  application  the  multiplexed  system  may  benefit  from  a 
redistribution  of  the  resolution  elements  via  a  discontinuous  FOV  or  an  extreme  FOV  aspect 
ratio.  In  the  detector-limited  case,  the  resolution  gain  of  the  multiplexed  system  may  even 
exceed  a  factor  of  N  if  the  multiplexing  assembly  provides  precise  dithering  that  enables 
super-resolved  sampling  as  demonstrated  in  [12]. 

Examination  of  the  detector  and  optical  cutoff  frequencies  of  the  modulation  transfer 
function  (MTF)  may  be  used  to  determine  if  the  system  is  detector  limited  or  optically 
limited.  For  square  pixels,  the  detector  MTF  curve  is  a  sine  function  for  which  we  define  a 
cut-off  frequency  at  its  first  zero 


M7F.. 


dp’ 


(11) 
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which  is  equal  to  twice  the  Nyquist  frequency.  The  optical  cutoff  frequency  for  incoherent 
illumination  of  a  circular  pupil  occurs  at 


MTF, 


cut, opt 


(12) 


To  first  order,  taking  the  ratio  of  Eqs.  (11)  and  (12)  provides  an  expression  for  the  optical 
blur  size  relative  to  the  detector  size  and  can  be  used  to  assess  whether  the  system  is  detector 
or  optically  limited  [8].  A  value  greater  than  2  implies  that  the  optics  MTF  drops  to  zero  at  or 
below  the  Nyquist  frequency,  thereby  defining  an  optically-limited  system.  A  multiplexed 
system  with  a  ratio  less  than  2  can  obtain  the  full  factor  of  N  increase  in  system  resolution 
relative  to  the  number  of  pixels  in  the  camera.  Note  that  the  N'^^  term  in  Eq.  (13)  applies 
specifically  to  division  of  aperture  systems  and  is  absent  if  the  full  pupil  area  is  divided 
amongst  the  multiplexed  channels. 


>  2,  optically  limited 


(13) 


2.2.3  SNR  and  dynamic  range 

The  power  in  each  channel  image  is  inversely  related  to  the  number  of  multiplexed  channels, 
as  described  in  Eq.  (3).  Translating  this  effect  into  a  channel  signal-to-noise-ratio  requires 
knowledge  of  the  encoding  method  and  the  scene  conditions.  In  some  cases  the  background 
noise  in  each  channel  will  be  the  sum  of  the  background  in  every  channel.  However,  this 
depends  on  whether  the  scene  is  signal  or  background  limited,  whether  it  is  densely  populated 
or  sparse,  and  whether  the  encoding  method  involves  periodic  attenuation  of  the  channels. 
Further,  the  signal  level  is  affected  by  the  relationship  between  the  pixel  size  and  the  point 
spread  function,  which  may  or  may  not  be  encoded  as  described  in  Section  3.4.  When 
selecting  the  number  of  multiplexed  channels  these  effects  should  be  evaluated  in  terms  of  the 
specific  sensing  application. 

In  the  case  where  each  channel  simultaneously  views  multiple  object  points  (i.e.  encoding 
by  means  other  than  channel  attenuation)  the  well-depth  of  the  pixel  is  shared  between  N 
channels.  In  sparse  signal-limited  applications  such  as  astronomical  observation  the  power  on 
each  pixel  may  result  predominantly  from  a  single  object  source  in  which  case  every  channel 
may  retain  the  full  dynamic  range  and  bit-depth  of  the  camera.  Conversely,  when  observing 
rich  scenes  or  in  background  limited  situations  the  dynamic  range  and  bit-depth  is  shared 
between  the  channels.  In  the  limiting  case  where  the  flux  on  each  pixel  is  uniformly 
distributed  between  the  channels  the  resulting  dynamic  range  (DRn)  and  bit-depth  (Bn)  of  the 
disambiguated  image  may  be  computed  by  Eqs.  (14)  and  (15): 


and 


DRn 


DK 

N  ’ 


(14) 


BN=Bo-^og^N, 


(15) 


where  DNo  and  Bo  are  the  dynamic  range  and  bit  depth  of  the  camera  used  for  image 
collection.  This  provides  a  simple  approximation  that  for  every  2  multiplexed  channels  the 
camera  output  is  reduced  by  1-bit. 

3.  Optical  design  considerations 

We  have  identified  a  number  of  optical  design  considerations  specific  to  optical  systems  that 
use  division  of  aperture  as  the  multiplexing  architecture.  In  this  section  we  describe  issues 
related  to  the  illumination  uniformity  of  the  multiplexed  image  that  are  influenced  by  the 
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design  of  the  multiplexing  assembly.  These  include  its  location  relative  to  the  entrance  pupil, 
pupil  aberrations,  and  vignetting.  Methods  of  dividing  the  pupil  by  refractive  and  reflective 
multiplexing  assemblies  are  also  described.  A  number  of  spatial  and  temporal  encoding 
schemes  are  discussed  along  with  methods  to  incorporate  them  into  the  optical  design. 

3. 1  Multiplexed  image  illumination  uniformity 

Division  of  aperture  presents  the  challenge  of  achieving  a  high  degree  of  illumination 
uniformity  across  the  FOV  of  each  channel.  In  addition  to  the  well  know  effects  of  image 
distortion  and  Cos'*  roll  off,  the  uniformity  of  each  channel’s  image  is  degraded  by  non- 
uniform  sampling  by  the  multiplexing  assembly.  Figure  2  illustrates  three  fundamental 
configurations  for  the  optical  system  with  a  notional  9-channel  multiplexing  assembly.  In 
panel  (a)  the  multiplexing  assembly  serves  as  an  aperture  stop  for  the  parent  lens.  In  principle, 
this  configuration  can  achieve  the  best  image  channel  illumination  uniformity  provided  that 
the  sampling  elements  are  sized  with  an  equal  area  and  the  multiplexing  assembly  is 
sufficiently  thin  to  fully  divide  the  pupil  near  the  plane  of  the  aperture  stop.  In  panel  (b)  of 
Fig.  2  the  multiplexing  assembly  is  positioned  at  a  remote  location  with  respect  to  the 
aperture  stop.  The  illumination  uniformity  of  each  channel  is  degraded  because  the  projection 
of  ray  bundles  onto  the  multiplexing  assembly  varies  across  the  FOV.  The  shift  of  the  center 
of  each  ray  bundle  with  respect  to  the  center  of  the  multiplexing  assembly  (6)  may  be 
expressed  as 

S  =  zi?Ln9,  (16) 

where  z  is  the  distance  from  the  entrance  pupil  to  the  multiplexing  assembly  and  0  is  the  field 
of  view  angle.  In  this  configuration  it  is  important  to  keep  5  small  with  respect  to  the  size  of 
the  sub-pupil  multiplexing  elements.  As  5  increases  differential  vignetting  is  observed  across 
the  fields  of  view  of  the  individual  channels. 

A  convenient  layout  uses  an  optical  design  that  projects  the  entrance  pupil  to  the 
multiplexing  assembly.  This  is  shown  notionally  with  a  reimaging  design  in  panel  (c)  of  Fig. 
2.  Pupil  aberrations  become  a  consideration  in  this  layout  because  they  can  distort  the  shape 
of  the  entrance  pupil  as  the  field  angle  increases  [9].  The  results  of  excessive  pupil  aberration 
are  multiplexing  image  non-uniformities  that  are  similar  to  channel-dependant  vignetting.  If 
necessary,  it  is  possible  to  reduce  pupil  aberration  in  an  optical  design  by  limiting  higher- 
order  aberrations  induced  by  the  elements  [10].  However,  this  limits  the  ability  to  balance  low 
order  aberrations  with  higher  orders,  which  may  require  additional  optical  elements  to 
achieve  satisfactory  image  quality. 
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Fig.  2.  Configurations  for  division  of  aperture  systems  and  their  beam  projections  on  the 
multiplexing  assembly  (MA).  (a)  the  MA  serves  as  the  aperture  stop,  (b)  a  remotely  located 
MA,  and  (c)  a  design  that  projects  the  entrance  pupil  to  the  MA  with  aberration  of  the  pupil. 

3.2  Multiplexing  assembly  design 

The  multiplexing  assembly  (MA)  may  consist  of  an  array  of  prisms  or  mirrors  to  direct 
multiple  fields  of  view  into  the  parent  lens.  The  selection  of  the  MA  architecture  has 
implications  on  a  number  of  system  properties  including  the  packaging  volume  and  encoding 
uniformity.  A  refractive  MA  has  the  benefit  of  compactness,  but  introduces  dispersion  and 
anamorphic  distortion  into  the  image.  A  reflective  assembly  will  typically  occupy  a  larger 
volume,  but  will  not  degrade  or  distort  the  image  formed  by  the  parent  lens.  Further 
considerations  include  the  effects  of  image  rotation  caused  by  the  MA  layout. 

Placement  of  the  MA  should  be  as  close  as  possible  to  the  aperture  stop  or  a  pupil  image 
to  achieve  good  illumination  uniformity.  Further,  if  planar  surfaces  are  used  the  MA  is  ideally 
located  in  a  collimated  space  where  an  angular  deviation  of  rays  will  not  induce  decentered 
aberrations  into  the  channels.  For  an  infinite-conjugate  lens  an  obvious  location  for  the  MA  is 
in  front  of  the  parent  lens,  which  can  expand  its  FOV  beyond  the  lens’  aberration  limit. 
Finite-conjugate  lenses  can  use  an  internal  MA  placed  in  a  collimated  space  within  the  lens 
assembly. 

3.2.1  Refractive  multiplexing  assemblies 

A  prism-based  MA  allows  for  compact  in-line  optical  designs.  The  prisms  can  be  easily 
placed  at  the  aperture  stop  or  entrance  pupil  of  the  optical  design  and  encoding  can  be 
performed  through  a  number  of  simple  methods  described  in  the  following  section.  Figure  3 
shows  notional  prism-based  MA  designs  that  divide  the  aperture  of  the  entrance  pupil  in  front 
of  the  lens. 
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Fig.  3.  (a)  a  notional  2-element  single-prism  MA  for  a  LWIR  camera.  Encoding  is  performed 
by  a  simple  motor  assembly  that  rotates  the  MA  about  the  optical  axis  of  the  lens,  (b)  optical 
layout  for  a  4-channel  multiplexing  assembly  based  on  achi‘omatic  prisms. 

The  chief  limitation  of  refractive  MAs  is  dispersion  caused  by  the  prisms,  which  can 
significantly  limit  the  extent  of  the  multiplexed  field  of  view.  By  expressing  a  system’s 
tolerance  for  chromatic  image  blur  in  terms  of  its  angular  resolution  a  practical  upper  limit  for 
MA  deviation  can  be  derived  as  follows.  We  begin  by  defining  the  dispersion  of  a  prism  in 
terms  of  its  Abbe  Number  (or  ‘V-number).  Equation  (17)  defines  the  V-number  for  a  material 
in  terms  of  its  refractive  index  («)  at  the  lower,  middle,  and  highest  wavelengths.  Table  1 
shows  a  selection  of  low  dispersion  materials  in  common  visible  and  infrared  wavebands. 
Using  the  thin  prism  approximation  (i.e.  small  apex  angle,  a)  the  angular  deviation  at  the 
center  of  the  waveband  (d)  and  the  dispersion  (d)  of  a  prism  may  be  approximated  by  Eqs. 
(18)  and  (19).  These  quantities  are  shown  in  Fig.  4. 


V  =  - 


—1 


(17) 


(18) 


A 


I 

V 


(19) 


The  field  of  view  of  an  individual  pixel  (^)  is  commonly  referred  to  as  the  instantaneous 
field  of  view  (IFOV)  and  defines  the  angular  resolution  of  the  system.  As  the  dispersion 
approaches  or  exceeds  the  IFOV  there  is  a  noticeable  loss  of  resolution.  The  IFOV  may  be 
approximated  as  the  ratio  of  the  pixel  pitch  to  the  effective  focal  length. 


(20) 


We  can  then  derive  a  maximum  deviation  angle  for  the  MA  in  terms  of  a  factor,  k,  multiplied 
by  the  IFOV,  where  k  is  the  allowance  for  chromatic  image  blur  relative  to  the  size  of  a  pixel. 
For  example,  a  k-factor  of  0.25  permits  a  maximum  chromatic  image  of  14  pixel  to  be  caused 
by  the  MA. 

(21) 
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Fig.  4.  Beam  deviation  and  dispersion,  (a)  a  thin  prism,  (b)  secondary  dispersion  in  an 
achromatic  prism  pair. 

Equations  (19)  through  (21)  can  then  be  combined  to  describe  a  maximum  deviation  angle 
(<5„,ax)  for  any  channel  of  the  MA  with  respect  to  the  optical  axis  of  the  parent  lens: 

S^=kV(p.  (22) 

The  value  kV  places  an  upper  limit  on  the  size  of  the  multiplexed  image.  For  example,  if  k 
=  0.25  and  V=  100  the  MA  can  only  divert  a  channel  by  25  IFOVs  with  respect  to  the  optical 
axis.  It  becomes  evident  that  a  single  prism  MA  is  insufficient  for  most  broadband  imaging 
applications  because  deviations  of  hundreds-to-thousands  of  IFOVs  are  required  to  fiilly 
separate  the  channels  in  object  space. 


Table  1.  A  selection  of  low-dispersion  materials. 


Waveband 

Material 

V-Number 

Visible  [486-656nm] 

CaFz 

N-FK51A 

95.2 

84.4 

SWIR  [900-1700nm] 

CaFz 

N-FK51A 

99.6 

76.2 

MWIR  [3-5  pm] 

Silicon 

236.5 

LWIR  [8-12  pm] 

Germanium 

783.2 

An  achromatic  prism  pair  can  greatly  extend  the  useful  range  of  a  refractive  MA. 
Analogous  to  an  achromatic  doublet,  an  achromatic  prism  pair  deviates  two  wavelengths  to 
the  same  angle  and  results  in  a  smaller  secondary  dispersion  (e)  as  shown  in  Fig.  4.  Secondary 
dispersion  can  be  expressed  in  terms  the  difference  between  the  V-Numbers  and  partial 
dispersions  {P)  of  the  two  elements  as 


e  = 


^ P  -P^ 
-*2  -*1 

V  -V 

v’^2  *^17 


(23) 


where. 


P  = 


'A,* 


(24) 


Then  for  a  secondary  dispersion  oie  =  k(p  we  derive  a  maximum  deviation  angle  of: 


V  -V 

r  2  Kj 


P  -P 

1  2  J  I 


k(p. 


(25) 
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The  leading  factor  of  Eq.  (25)  is  shown  in  Table  2  for  a  selection  of  achromatic  prism 
pairs  with  low  secondary  dispersion.  The  data  show  a  vast  improvement  in  the  maximum 
deviation  angle  which  indicates  that  a  refractive  MA  can  multiplex  widely  spaced  FOVs 
without  an  appreciable  resolution  loss. 

Table  2.  Examples  of  achromatic  prism  pairs  with  small  amounts  of  secondary 

dispersion. 


Waveband 

Material 

[V2-V,]/[P2-Pl] 

Visible  [486-656nm] 

N-FK51A 

N-PSK3 

3.27E  +  04 

SWIR  [900-1700nm] 

CaFz 

N-BAFIO 

5.88E  +  04 

Silicon 

3.12E  +  04 

MWIR  [3-5  urn] 

Germanium 

Germanium 

4.16E  +  03 

LWIR[8-12  nm] 

ZnSe 

Another  consideration  for  refractive  multiplexing  assemblies  is  anamorphic  distortion. 
The  deviation  and  dispersion  equations  in  this  section  were  simplified  using  the  small  angle 
approximation  for  ‘thin  prisms’.  As  the  prism’s  apex  angle  increases  for  larger  deviations  the 
deviation  angle  becomes  more  dependent  on  the  angle  of  incidence  into  the  prism.  With  an 
extended  field  of  view  ray  angles  are  distorted  producing  a  warped  image.  Anamorphic 
distortion  can  be  minimized  through  the  optical  design  of  the  MA  and  corrected  in  the  final 
image  through  calibration  and  post-processing;  however,  significant  levels  of  distortion  may 
be  unavoidable  for  large  deviation  angles  and  may  alter  the  angular  resolution,  projected  pupil 
area,  and  image  illumination  as  a  function  of  field  of  view  angle. 

3.2.2  Reflective  multiplexing  assemblies 

A  multiplexing  assembly  consisting  of  an  array  of  mirrors  has  the  advantage  of  unlimited 
deviation  angles  without  degrading  the  image  quality  through  dispersion  or  distortion.  It  can 
also  utilize  the  pupil  area  more  efficiently  because  the  mirror  elements  can  be  mounted  from 
behind  or  incorporated  into  a  single  monolithic  element  with  knife  edge  transitions  between 
the  facets.  The  disadvantages  of  this  architecture  are  that  it  typically  occupies  a  larger  volume 
than  a  refractive  MA,  and  the  MA  must  be  tilted  with  respect  to  the  optical  axis  of  the  lens.  If 
the  MA  is  tilted  with  respect  to  the  aperture  stop  or  pupil,  illumination  non-uniformities  will 
also  increase. 

Figure  5  shows  reflective  MA  layouts  that  the  authors  have  found  useful.  Panel  (a)  shows 
a  single  element  MA  placed  in  front  of  a  narrow  field  of  view  lens.  The  MA  is  tilted  with 
respect  to  the  optical  axis  and  can  be  modeled  as  either  a  remote  aperture  stop  or  a  vignetting 
surface.  Panel  (b)  shows  a  multi-element  MA.  This  architecture  uses  an  additional  fold  mirror 
that  allows  for  beam  steering  and  can  be  used  to  increase  the  physical  separation  between 
channels  for  encoding  or  depth  imaging  applications. 


(a)  (b) 

Fig.  5.  Notional  Reflective  multiplexing  assemblies,  (a)  a  narrow  FOV  lens  used  with  a  single 
element  MA  acting  as  a  remote  tilted  aperture  stop,  and  (b)  a  multi-element  MA  shown  with  a 
reimaging  lens  that  projects  the  pupil  to  the  MA. 
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1.1  Image  rotation 

The  multiplexing  assembly  may  introduce  rotation  or  parity  changes  in  the  image.  This  occurs 
when  multiplexing  elements  divert  light  at  angles  between  the  horizontal  and  vertical  planes 
of  the  detector  array,  or  with  an  odd  number  of  reflections,  respectively.  The  optical  design  of 
the  MA  can  be  used  to  control  the  image  rotation  to  some  degree,  and  additional  elements 
may  be  introduced  to  deterministically  control  the  image  rotation  to  a  prescribed  value.  The 
consequences  of  image  rotation  introducing  gaps  or  overlap  between  adjacent  FOV  channels 
should  be  considered.  In  some  cases  this  rotation  can  be  used  as  a  method  of  passive  image 
encoding  as  described  in  the  following  section. 

4.  Image  encoding  and  decoding  methods 

Image  reconstruction  from  multiplexed  measurements  requires  per  channel  image  encoding 
and  a  subsequent  decoding  procedure.  Notable  exceptions  to  this  requirement  are  when 
imaging  a  scene  from  a  constrained  domain  in  such  applications  as  pattern  recognition  or  star 
tracking,  or  when  multiple  multiplexed  systems  observe  the  same  scene  [11].  Encoding  and 
decoding  methods  can  be  split  into  two  fundamental  categories:  spatial  and  temporal. 

4. 1  Spatial  image  encoding 

Spatial  encoding  is  most  effective  when  observing  a  sparse  scene  of  point-like  sources  such  as 
the  night  sky  or  in  unresolved  detection  and  tracking  applications.  If  the  point  spread  function 
of  each  charmel  is  uniquely  spatially  encoded  and  the  scene  is  sparse  (potentially  after  some 
operation  such  as  background  subtraction  is  applied)  a  sparse  approximation  algorithm  can 
robustly  recover  the  sparse  image.  In  this  approach,  we  assert  that  the  signal  is  sparse  in  an 
over-complete  dictionary  formed  from  multiple  matrices,  where  each  matrix  is  a  convolution 
with  a  single  charniel’s  point  spread  function.  A  Matching  Pursuit  [13]  algorithm  can  find  a 
sparse  approximation  for  the  measured  signal  in  this  dictionary.  There  are  a  number  of 
alternative  sparse  approximation  approaches  that  can  be  used,  based  on  LI  regularization  as 
discussed  in  [14,  15],  message  passing  in  graphical  models  [16],  and  extensions  of  matching 
pursuit  [17]. 

Various  methods  of  PSF  engineering  can  be  applied  to  encode  the  image  including  using 
diffractive  or  holographic  elements  or  by  applying  a  low  order  aberration  term  to  elements  in 
the  multiplexing  assembly.  Aberrations  in  the  parent  lens  itself  may  be  sufficiently  unique  to 
encode  the  image  in  a  division  of  aperture  system,  but  variation  across  the  field  of  view  may 
require  a  spatially  varying  matched  filter.  Another  spatial  encoding  method  involves  dividing 
the  transmission  of  each  channel  into  a  pattern  of  two  or  more  image  points.  Splitting  the  PSF 
into  two  image  points  is  the  most  signal-efficient  method  of  spatial  image  encoding  and  can 
be  accomplished  by  such  methods  as  using  a  weak  birefringent  prism  or  some  other  weakly 
deviating  dichroic  or  intensity  splitting  element. 

4.2  Temporal  image  encoding 

In  conventional  photographic  and  video  imaging  applications  that  are  densely  populated  with 
extended  objects  temporal  image  encoding  methods  may  be  used  to  disambiguate  a 
multiplexed  image.  Temporal  encoding  may  be  accomplished  by  applying  a  series  of  image 
shifts,  intensity  modulations,  or  time-varying  point  spread  functions  to  the  channels.  Active 
methods  of  implementation  include  tilting  or  rotating  elements  in  the  multiplexing  assembly, 
applying  a  variable  attenuation  or  shuttering  to  a  sub-set  of  the  charmels,  or  using  a  spatial 
light  modulator  to  actively  encode  the  point  spread  function.  Passive  temporal  encoding  can 
be  achieved  if  the  multiplexing  assembly  imparts  a  unique  rotation  to  each  channel  and  a 
known  relative  motion  exists  between  the  sensor  and  the  scene.  In  this  case,  observing  the 
trajectory  of  features  in  the  multiplexed  image  as  the  sensor  or  scene  moves  in  a  known  way 
provides  the  method  of  disambiguation. 
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One  method  of  image  decoding  is  to  construct  a  fully  determined  set  of  equations  from  a 
sequence  of  N  uniquely  shifted  measurements.  This  process  is  described  below  as  for  an 
encoding  method  that  involves  known  2-dimensional  per-channel  image  shifts  in  each 
measurement.  Results  obtained  using  this  approach  are  shown  in  Section  6.  We  begin  by 
expressing  an  imaging  process  as  a  linear  transformation  from  object  to  image  space  with 
additive  noise  as 


z  =  Ax +  £, 


(26) 


where  ze  is  the  measured  image  observed  on  the  focal  plane,  A  e  R^™  is  the  imaging 
transformation  matrix,  xe  R'"  is  a  discretized  m-pixel  representation  of  the  scene,  and 
£■  E  R^  represents  the  noise  corrupting  each  pixel  measurement.  A  multiplexing  imaging 
process  has  a  transformation  matrix  comprised  of  an  encoding,  a  selection,  a  downsampling, 
and  a  multiplexing  operation.  Thus,  the  multiplexing  imaging  process  can  be  written  as 


Z  —  v4  y4  v4  y4  X  ~i~  £ 

'^multiplex  '^downsample  '^selection  '^encoding 


(27) 


The  encoding  operation,  e  K"™  ^  produces  an  encoded  version  of  the  underlying 

scene  for  each  of  the  N  channels.  In  this  example  the  encoding  matrix  produces  per-channel 
image  shifts  of  an  integer  number  of  pixels  in  the  reconstructed  image  space.  The  selection 
matrix,  ^  ,  represents  the  mapping  from  the  shifted  scene  coordinates  to  focal 

plane  coordinates.  The  downsampling  factor,/  is  the  ratio  of  the  area  of  a  focal  plane  pixel  to 
that  of  the  reconstructed  pixel,  and  fl  <  m  .  Downsampling,  e  ,  is  the  re¬ 

sampling  from  a  higher  resolution  reconstructed  image  to  that  of  the  lower  resolution  focal 
plane.  When /=  1,  the  resolutions  are  matched  and  Adownsampk  is  simply  an  identity  matrix. 
Finally,  the  multiplexing  operation,  e  R““ ,  sums  over  the  N  channels  which  are 

physically  superimposed  on  the  I  pixel  focal  plane.  In  general,  solving  for  x  given  z  is  an  ill- 
posed  problem  as  I  <  m  due  to  the  multiplexing  and  downsampling  operations.  However, 
taking  p  images  of  a  static  scene  with  different  image  shifts  can  be  written  as 

z  =  Ax  +  £  (28) 


A 

Where  z  = 

m 

II 

A 

G  R"'™  ,  and  f  = 

A_ 

e  R'’/  If  pi  >m  then  A  can  be  full 


rank  for  appropriately  chosen  shift  matrices,  which  allows  for  solving  for  the  scene  by 
inverting  A  , 


X  =  A  ’z. 


(29) 


If  pi  <  m  then  A  is  undetermined,  but  with  properly  chosen  shifts  it  contains  sufficient 
structure  to  recover  a  restricted  class  of  signals  if  the  proper  regularization  is  used.  For 
example,  if  the  signal  is  sparse  in  some  basis,  then  the  image  can  be  recovered  using  standard 
nonlinear  estimators  used  in  compressed  sensing  [14,  15]. 

For  large  images  (e.g.  a  multi-megapixel  image)  the  A  matrix  can  become  so  large  that  a 
direct  inverse  is  impractical,  since  computing  an  inverse  scales  cubically  with  the  number  of 
elements  in  the  image.  However,  A  is  inherently  sparse  since  a  shift  corresponds  to  a  sparse 
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matrix,  enabling  a  reduction  in  computational  cost.  By  modeling  the  A  and  operations 
(without  having  to  explicitly  compute  the  matrices)  we  can  use  an  iterative  solver  such  as 
LSQR  [18]  that  can  approximate  rather  than  directly  compute  the  inverse. 

5.  Experimental  prototypes 

In  this  section,  we  present  test  results  for  prototype  multiplexed  sensors  in  a  star  tracking 
application.  Two  experimental  sensors  were  constructed  that  used  identical  shortwave 
infrared  (SWIR,  0.9- 1.7  pm  waveband)  cameras  (Goodrich  SU640KTS)  and  50  mm  SWIR 
lenses.  Similar  5-channel  multiplexing  assemblies  were  used  for  both  sensors.  Figure  6  shows 
the  design  of  the  multiplexing  assembly  and  the  resulting  rotated  field  of  view  arrangement. 
The  pupil  dividing  element  of  the  multiplexing  assembly  was  a  custom  diamond-turned  5- 
surface  monolithic  faceted  reflector  with  knife  edge  transitions  between  the  facets  to 
maximize  pupil  sampling  efficiency.  This  element  was  made  smaller  than  the  pupil  diameter 
of  the  lens  and  placed  as  close  as  possible  to  the  lens  to  mitigate  image  irradiance  non¬ 
uniformities  caused  by  vignetting  as  illustrated  in  Fig.  2(b).  A  3D-printed  element  provided 
the  mounting  structure  for  the  multiplexing  assembly.  The  two  prototype  configurations 
differed  in  the  method  of  encoding  and  the  design  of  the  fold  mirror  element  of  the  MA. 

In  SWIR  prototype  configuration  I  a  simple  metalized  fold  mirror  was  used  and  a  6- 
position  filter  wheel  was  placed  between  the  pupil  dividing  element  and  the  lens.  The  filter 
wheel  consisted  of  an  open  aperture,  and  5  unique  aperture  patterns  each  of  which  blocked  1 
channel.  The  open  configuration  allowed  for  experiments  using  single  frames  of  the  5- 
channel  multiplexed  image,  and  for  passive  temporal  encoding  experiments  via  rotated  fields 
of  view  with  relative  scene  motion.  The  remaining  5  filter  positions  allowed  for  temporal 
encoding  experiments  in  which  a  sequence  of  images  could  be  collected  with  a  single  channel 
attenuated. 

-40  Azimuth  (dea)  40 

10 

'SS 

"O 

c 
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Fig.  6.  A  five-channel  optically  multiplexed  sensor  and  field  of  view  projection. 

SWIR  prototype  configuration  2  used  a  wedged  dichroic  fold  mirror  element  for  spatial 
encoding  experiments.  A  commercially  available  dichroic  window  (Thor  labs  DMSP1500L) 
was  placed  in  front  of  a  metalized  reflector  with  a  small  tilt  between  the  elements.  Each 
channel  created  a  two-point  image  PSF  by  reflecting  wavelengths  above  -1.5  pm  from  the 
first  surface  and  wavelengths  below  -1.5  pm  from  the  second  surface  at  a  slightly  different 
angle.  Rotation  of  this  fold  mirror  assembly  allowed  for  the  two-point  PSF  orientation  to  be 
uniquely  rotated  for  each  channel.  The  exact  wavelength  of  the  transition  varied  from  channel 
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to  channel  because  the  band  edge  shifted  with  angle  of  incidence.  Figure  7  shows  the 
prototype  systems  along  with  a  visible  witness  camera. 


Fig.  7.  SWIR  five-channel  prototypes.  Left:  SWIR  prototype  2  with  a  wedged  dichroic  fold 
mirror  for  spatial  encoding.  Center:  a  visible  witness  camera.  Right:  SWIR  prototype  1  with  a 
metalized  fold  mirror  and  filter  wheel. 

Active  temporal  encoding  experiments  were  performed  using  a  six-channel  visible 
wavelength  prototype  described  in  [12].  The  multiplexing  assembly  was  constructed  from  an 
array  of  six  mirrors  mounted  on  independent  piezeoelectic  tip/tilt  platforms.  A  200  mm  parent 
lens  (Nikon  AF-S  NIKKOR  200  mm  F/2G  ED  VR  11)  was  used  with  a  4.2  megapixel  camera 
(Point  Grey  GS3-U3-41C6M-C)  to  produce  six  3.2°x3.2°  channels.  Mirror  sections  were 
sized  to  equally  divide  the  pupil  area  at  an  F/4  aperture  setting.  This  produced  channels  with 
an  effective  F/#  of  F/9.8.  This  architecture  represents  an  embodiment  of  a  single  reflective 
multiplexing  assembly  as  shown  in  Fig.  5,  with  the  multiplexing  assembly  serving  as  a  remote 
aperture  stop  as  shown  in  Fig.  2(a).  The  visible  prototype  allowed  for  temporal  encoding  and 
super-resolution  experiments  using  rapid  and  precise  image  shifting.  Spatial  encoding  could 
also  be  performed  with  this  architecture  by  rapidly  sweeping  the  mirrors  during  the  camera’s 
integration  period  to  encode  the  PSF  with  motion  blur.  Figure  8  shows  the  visible  prototype. 

A  summary  of  system  parameters  for  the  prototypes  and  their  conventional  parent  lenses 
is  provided  in  Table  3. 

Table  3.  Prototype  system  parameters 


System 

Encoding  Method 

#  Mnltiplcxed 
Channels 

Focal  Length 

mm 

F/# 

FOV 

Conv.  SWIR  Lens 

- 

- 

50 

2 

18“xl4° 

SWIR  Prototype  1 

Shutter  /  Rotation 

5 

50 

4.5 

~80“xl4“ 

SWIR  Prototype  2 

PSF  Encoding 

5 

50 

4.5 

~80°xl4“ 

Conv.  Vis.  Lens 

- 

- 

200 

(2)4 

3.2“x3.2° 

Visible  Prototype 

Image  Shifting 

6 

200 

9.8 

19“x3.2“ 
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Fig.  8.  Six-channel  visible  prototype. 

6.  Experimental  demonstration 

Sparse-scene  experiments  were  performed  using  the  SWIR  prototypes  to  explore  different 
methods  of  eneoding  and  scene  reconstmction.  Observations  the  night  sky  were  performed  to 
demonstrate  that  multiplexed  images  could  be  used  for  point  source  detection,  which  can 
enable  astronomical  navigation  and  attitude  detennination  by  star  tracking.  The  motivation  of 
this  experiment  was  to  demonstrate  and  validate  our  aperture  division  and  encoding  concepts. 

Multiplexed  images  from  the  SWIR  prototypes  are  shown  in  Fig.  9.  Panel  (a)  shows 
results  from  prototype  2.  Dichroic  splitting  of  the  incident  light  produces  a  double  image  of 
each  star.  Each  channel  introduces  a  unique  rotation  orientation  between  the  two  points, 
which  is  used  for  disambiguation.  A  reconstruction  of  the  sky  from  prototype  2  measurements 
is  shown  in  panel  (d),  and  was  obtained  by  matched  fdtering  of  the  multiplexed  scene  with 
the  known  point  spread  function  of  each  channel.  Passive  and  active  temporal  encoding  was 
demonstrated  by  prototype  1.  Panel  (b)  shows  a  long  exposure  image  captured  through  the 
open  aperture  of  the  fdter  wheel.  As  the  stars  swept  across  the  sky  different  streak 
orientations  were  observed  in  the  image,  which  was  used  to  indicate  the  channel  position  of 
each  star.  This  demonstrated  a  method  of  passive  temporal  encoding  through  known  scene 
motion.  A  method  of  active  temporal  encoding  is  shown  in  panel  (c)  of  Fig.  9.  In  this 
experiment,  the  fdter  wheel  was  cycled  through  5  positions  during  a  long  exposure  image. 
Each  position  of  the  fdter  wheel  blocked  a  single  channel.  The  streaked  image  of  each  star 
showed  a  discontinuity  indicating  the  channel  from  which  it  originated. 
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(a)  PSF  Encoding 


(b)  Rotation  Encoding 


(c)  Shutter  Encoding 


(d)  De-multiplexed  Stars 


Azimuth  (degrees) 


Fig.  9.  Multiplexed  images  of  the  night  sky  and  a  disambiguated  image,  (a)  multiplexed  image 
from  SWIR  prototype  2,  (b)  a  long  integration  multiplexed  image  from  SWIR  prototype  1 
demonstrating  passive  temporal  encoding  via  image  rotation,  and  (c)  a  multiplexed  image  from 
SWIR  prototype  1  demonstrating  active  temporal  encoding,  (d)  a  reconstruction  of  the  night 
sky  prototype  2. 

Dynamic  image  shifting  with  the  visible  prototype  allowed  for  reconstruction  of  an 
arbitrary  infonnation-rich  scene.  Figure  10  shows  data  collected  with  the  visible  prototype. 
The  mirrors  in  the  multiplexing  assembly  were  arranged  to  cover  a  six-channel  19°x3.2°  field 
of  view  with  a  small  overlap  between  the  channels  to  aid  the  image  reconstruction.  A 
multiplexed  image  is  shown  in  Fig.  10(a).  Panel  (b)  shows  a  24  megapixel  image  obtained  by 
inverting  the  matrix  described  in  Section  4.2. 


Fig.  10.  Images  collected  with  the  visible  waveband  prototype,  (a)  a  single  four  megapixel 
multiplexed  image,  (b)  a  24  megapixel  image  reconstructed  from  a  sequence  of  uniquely 
encoded  multiplexed  frames. 
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7.  Conclusion 

This  paper  has  described  new  design  architectures  for  optically  multiplexed  imaging  systems 
with  emphasis  on  the  division  of  aperture  implementation.  Considerations  relating  to  the 
initial  selection  of  channels  in  the  multiplexed  image  and  the  design  of  the  multiplexing 
assembly  have  been  provided  to  inform  the  design  of  future  systems.  Experimental  results 
were  presented  using  two  optically  multiplexed  SWIR  waveband  prototypes  each  consisting 
of  5  multiplexed  channels.  A  third  visible  waveband  prototype  used  6  imaging  channels, 
which  to  the  authors’  knowledge  is  the  highest  degree  of  multiplexing  performed  to  date. 
Efficient  techniques  for  spatial  and  temporal  encoding  have  been  described.  It  has  been 
demonstrated  that  optically  multiplexed  systems  can  be  used  to  reconstruct  sparse  night  sky 
scenes,  and  to  accurately  reconstruct  images  of  dense  scenes  via  temporal  encoding. 

Practical  division  of  aperture  systems  can  be  developed  with  a  high  number  of 
multiplexed  channels.  The  trade-off  of  aperture  area  for  extended  field  of  view  provides  a 
new  dimension  in  optical  design  by  allowing  narrow  FOV  lenses  to  perform  wide  FOV  tasks. 
Further,  by  decoupling  the  shape  of  the  observed  FOV  from  the  shape  of  the  detector 
multiplexing  enables  new  high  aspect  ratio  or  discontinuous  fields  of  view.  It  is  the  authors’ 
belief  that  optically  multiplexed  imaging  systems  have  a  high  potential  for  many  applications 
including  navigation,  machine  vision,  optical  communication,  photography,  and  surveillance. 
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